Skip to content

QML Load Snapshot Signet (160,000 height) #424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

D33r-Gee
Copy link
Contributor

@D33r-Gee D33r-Gee commented Oct 11, 2024

Open for review and testing only. Do not merge!

GUI Integration for UTXO Snapshot Loading

Overview

This PR adds initial GUI support for loading a signet UTXO snapshot at height 160,000, building on Bitcoin Core's assumeutxo infrastructure.

What This PR Does

  1. Adds a basic GUI interface for loading UTXO snapshots via the Connection Settings panel
  2. Implements non-blocking snapshot loading via QT Thread
  3. Provides visual confirmation of snapshot activation
  4. Adds a snapshot progress notifications accessible via a handler

Implementation Details

Core Components Modified

  1. Node Interface (src/node/interfaces.cpp)
  • Implements snapshotLoad() based on the loadtxoutset RPC
  • Uses background QT thread for non-blocking operation
  1. QML Integration (src/qml/models/nodemodel.cpp)
  • Manages snapshot loading state and progress
  • Introduces a worker class to manage snapshot metadata
  • Uses Qt's thread system to prevent UI blocking
  • Provides progress updates via Qt signals
  1. UI Components (src/qml/components/SnapshotSettings.qml)
  • File selection dialog
  • Snapshot verification status
  • Progress tracking and visualization

Key Design Decisions

  1. Non-blocking Implementation
  • Snapshot loading runs in separate QT thread
  • UI remains responsive during load
  1. Extensibility
  • Interface designed to accommodate future assumeutxo changes
  • Error handling framework in place

Testing Instructions

  1. Build
  2. Launch Bitcoin Core QML GUI
  3. Navigate to Connection Settings (after onboarding is done)
  4. Test with sample snapshot:
magnet:?xt=urn:btih:9da986cb27b3980ea7fd06b21e199b148d486880&dn=utxo-signet-160000.dat
  1. Verify (see screenshots below):
    • "Snapshot Loaded" confirmation appears
    • Block clock shows ~50% progress
    • For errors, check debug.log for [snapshot] or [loadsnapshot] messages
Ubuntu 22.04 Screenshots

Screenshot 2024-10-11 100308
Screenshot 2024-10-11 100319
Load_UTXO_Snapshots_FileDialog
Screenshot 2024-11-13 144609
Screenshot 2024-11-13 144917
Screenshot 2024-11-13 145350

Expected Behavior

  • File selection dialog works
  • Success/failure state properly displayed
  • Node becomes usable while background validation continues

Future Work

  1. Error Handling
  • Add user-facing error messages
  • Implement recovery paths
  • Add snapshot validation UI once background validation has completed
  1. Integration
  • Coordinate with upstream assumeutxo changes
  • Add support for network-specific snapshots (testnet and mainnet)
  • Integrate with initial setup wizard

Notes for Reviewers

Snapshot Compatibility

  • Load option available after Onboarding in the Connections Settings page
  • Currently only works with signet UTXO snapshot
  • Limited to specific 160,000 height
  • Validates against chain parameters m_assumeutxo_data
    (the changes there can be discarded once synced since m_assumeutxo_data is already hardcoded)

feedback welcome on the approach and implementation details.

@D33r-Gee D33r-Gee marked this pull request as draft October 11, 2024 17:11
@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from cb409e3 to 670ca05 Compare October 15, 2024 17:43
@D33r-Gee
Copy link
Contributor Author

With cb409e3 rebased and got rid off the extra options_mode settings which are not relevant for this evaluation.

Also deleted the extra loadsnapshot.cpp file and instead added the load snapshot function in src/node/interfaces.cpp

@yashrajd
Copy link

Should I be getting a binary specific for this issue? I tried loading a signet snapshot into the one for #421 but was unable to...

@D33r-Gee
Copy link
Contributor Author

Should I be getting a binary specific for this issue?

Yes please download the artifact for your system here

I tried loading a signet snapshot into the one for #421 but was unable to...

That's correct behaviour, 421 is UI only and doesn't not have backend wiring

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 670ca05 to b77f7c1 Compare October 16, 2024 19:53
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Oct 16, 2024

With b77f7c1 minor tweaks to the src/node/interfaces.cpp snapshotload() function.

It mimics the rpc call loadtxoutset from bitcoin/bitcoin repository

Next will connect the loading to the progress bar

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from b77f7c1 to c59e557 Compare October 22, 2024 21:57
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Oct 22, 2024

with c59e557 Integrated the UI with the C++ backend, implementing a progress callback function to dynamically update the loading progress based on the number of coins processed.

Touched src/validation.h and src/validation.cpp to extract the coin loading progress.

Next: will update the "Loaded Snapshot" stack with the real info from the snapshot

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from c59e557 to e3f8c70 Compare October 25, 2024 20:42
@D33r-Gee
Copy link
Contributor Author

With e3f8c70 moved the load snapshot functionality from nodeModel to optionsModel since it make it easier to store the snapshot info and reduces redundant variable declarations.

Copy link
Member

@jarolrod jarolrod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With e3f8c70 moved the load snapshot functionality from nodeModel to optionsModel since it make it easier to store the snapshot info and reduces redundant variable declarations.

e3f8c70 is a regression because optionsmodel is not an appropriate place for this, and we are now violating the role of settings.json to store the state of a process.

NodeModel and AppMode are more appropriate places for this information. But the entire approach here is still a concept NACK as we are performing unecessary changes to validation and chainparams.

@yashrajd
Copy link

yashrajd commented Oct 29, 2024

Ok I tested artefact#235 and functionality & UI seems to work.

Didn't see block clock so couldn't verify easily. Adding some UI feedback below:

assumeutxo-424-sc1 copy

  • fix list item width

assumeutxo-424-sc2 copy

assumeutxo-424-sc4 copy

  • fix list item width
  • fix icon size

assumeutxo-424-sc3 copy

  • blockHeight is accurate
  • the date in the copy above it seems right
  • didn't check hash accuracy
  • hash value is overflowing as you can see, I'd probably wrap it over 2-3 lines, perhaps add copy functionality

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from e3f8c70 to 90dc0dd Compare October 31, 2024 22:09
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Oct 31, 2024

with 90dc0dd rebased over main and addressed @jarolrod concerns by reverting back to just a minimal integration of the loading snapshot functionality.

@yashrajd I will address the UI issues next

As a follow up will investigate how to access the snapshot verification progress

Also looking into how to make the snapshot data persist and be available for display (i.e. snapshot block height, date, and merkle hash)

@D33r-Gee
Copy link
Contributor Author

But the entire approach here is still a concept NACK as we are performing unecessary changes to validation and chainparams.

For the chainparams the changes there are only temporary for evaluating and testing within this repo. I checked upstream and the signet snapshot has already been hardcoded so once our repo is synced the changes in chainparams.cpp can be discarded.

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 90dc0dd to 85592f5 Compare November 1, 2024 18:41
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Nov 1, 2024

with 85592f5 made the snapshot activation persist on node restart by making use of the interface with chain->hasAssumedValidChain

Also changed the green-check icon size to 30 pix

Next looking into how to extract the progress from the ActivateSnapshot process

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 85592f5 to 955c65d Compare November 4, 2024 22:39
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Nov 4, 2024

with 955c65d modified the snapshotLoad() function in src/node/interfaces.cpp and the initializeSnapshot() function in nodemodel.cpp to extract progress information using regular expressions.

This design decision aimed to keep snapshot loading mechanisms self-contained.

However, it looks like the use of std::stod is triggering some lint failures... will look into how to address that.

Perhaps the way to go is to use the handler in src/node/interfaces_ui ... will explore that path then update

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 955c65d to 0eba513 Compare November 8, 2024 18:35
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Nov 8, 2024

with 0eba513 reverted back to minimal snapshot load functionality.

Snapshot loading progress is still not were I want it to be so in the meantime replaced the progress bar with placeholder text.

Also updated the snapshot info display. once successfully loaded, and I'm using the hardcoded info in chainparams since if the snapshot has loaded successfully it means the info there has been validated.
This is make the info persistent and not rely on the snapshot.dat file (in case it gets deleted)

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 0eba513 to cd2ec69 Compare November 12, 2024 21:30
@D33r-Gee
Copy link
Contributor Author

With cd2ec69 added a section to keep track of snapshot loading, it is not ideal so will look for better ways to achieve it.

Since the core functionality is there will ask for "pre-reviews" before un-drafting

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch 2 times, most recently from 5d0756c to 262bb03 Compare November 13, 2024 21:23
@D33r-Gee
Copy link
Contributor Author

with 5d0756c updated SnapshotSettings.qml with comments documenting the loading process.

with 262bb03 updated nodemodel by adding the m_snapshot_loading bool so that the "Snapshot Loading" page persist if the user navigates away from it.

Next looking into how to finesse the snapshot loading progress...

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 109cf42 to cad966e Compare March 18, 2025 17:43
@D33r-Gee
Copy link
Contributor Author

with cad966e rebased over main.

Keeping this PR as draft until we get closer to the sync with upstream...

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from cad966e to 6c25cd7 Compare April 24, 2025 20:55
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented Apr 24, 2025

with 6c25cd7 :

Will keep this as draft until we get closer to the sync with upstream

@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented May 2, 2025

4e43df9 updates the following:

  • Introduce isIBDComplete bool in qml/nodemodel
  • Update ConnectionsSettings.qml to reflect the change

Once the CI completes will undraft since we are not exactly sure when the sync with upstream will happen. And this feature will be demoed at upcoming events.

Feedback welcomed!

@D33r-Gee D33r-Gee marked this pull request as ready for review May 2, 2025 20:04
@D33r-Gee D33r-Gee changed the title [DRAFT] WIP QML Load Snapshot Signet (160,000 height) QML Load Snapshot Signet (160,000 height) May 2, 2025
@D33r-Gee D33r-Gee requested review from GBKS and johnny9 May 5, 2025 14:50
@D33r-Gee D33r-Gee requested a review from yashrajd May 7, 2025 14:14
@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from 4e43df9 to bbbbf42 Compare May 9, 2025 18:17
@D33r-Gee
Copy link
Contributor Author

D33r-Gee commented May 9, 2025

With bbbbf42 adds error page to handle when a invalid snapshot gets used, this is based on @GBKS feedback in #449

Feedback welcomed!

@hebasto
Copy link
Member

hebasto commented May 12, 2025

Based on #421.

This is confusing, given that #421 is closed.

@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from bbbbf42 to e02f382 Compare May 15, 2025 16:36
@D33r-Gee
Copy link
Contributor Author

e02f382 rebased over main

@D33r-Gee
Copy link
Contributor Author

friendly ping @yashrajd and @johnny9

D33r-Gee added 3 commits May 26, 2025 09:44
- This introduce the UI flow to load a AssumeUTXO snapshot
- It modifies the connection settings
- Adds a SnapshotSettings file, Icon, and modified progress
bar.
- Also it adds error page on snapshotloading failure
- Extend node interface with virtual functions for UTXO snapshot loading
- Add signal mechanism to monitor snapshot loading progress
- Include predefined signet UTXO dataset in chainparams for validation
- Add SnapshotQml class to encapsulate snapshot loading operations
- Integrate QML properties for UTXO snapshot management
- Wire up snapshot loading functionality with node interface
@D33r-Gee D33r-Gee force-pushed the qml-load-snapshot-signet branch from e02f382 to d2fccc4 Compare May 26, 2025 16:44
@D33r-Gee
Copy link
Contributor Author

with d2fccc4 rebased over main

@yashrajd
Copy link

friendly ping @yashrajd and @johnny9

I will try to test this today.

@yashrajd
Copy link

ok I just tested the latest build on signet...following are observations:

tl;dr – snapshot loading worked!

(but shown hash doesn't seem to match file hash from terminal (but maybe I'm barking up the wrong tree)

assumeutxo testing


  • I see the block status widget updated after snapshot loading is completed.

after loading snapshot


  • then I quit the app, and relaunched it....the block status widget not on the same page anymore...

relaunced app after loading


  • go to Connection Settings, the Load snapshot option has disappeared (iirc this is expected behaviour) so I assume it still worked.

load snapshot option has disappeared

Did I test it right? Did I miss something I should've tested...lemme know.

@D33r-Gee
Copy link
Contributor Author

(but shown hash doesn't seem to match file hash from terminal (but maybe I'm barking up the wrong tree)

Thanks @yashrajd for the attached screenshot showing the two different hashes associated with the loaded UTXO snapshot.

I’d like to clarify the distinction between them to help avoid any confusion:

  1. SHA-256 Hash of the Snapshot Data File

    • This hash is generated by running a command such as shasum -a 256 utxo-signet-160000.dat directly on the snapshot file.

    • Example from the image:

      eeeca845385ba91e84ef58c9d38f98f246a24feadaad57fe1e5874f3f92ef8c  utxo-signet-160000.dat
      
    • It represents a checksum of the full file contents and is typically used to verify file integrity during download or transfer.

  2. Hash Displayed in the Application (Snapshot Metadata Hash)

    • The hash shown in the user interface (e.g., 5225141cb62dee63ab3be95f9b03d60801f264010b1816d4bd00618b2736e7be) is not the SHA-256 of the file itself.
    • Rather, it’s the hash of the serialized snapshot metadata, generated internally using the hash_serialized function after parsing the file.
    • This metadata hash is what Bitcoin Core uses to validate the snapshot’s correctness and ensure compatibility with the expected chain state. You can find relevant references in the source code, such as AssumeutxoHash in chainparams.cpp and ExpectedAssumeutxo in validation.cpp.

I hope this helps clarify the purpose and origin of each hash. Please let me know if the above makes sense?

@D33r-Gee
Copy link
Contributor Author

Here's some TLDR answers to @yashrajd experience (let me know if you'd like more details) Also let me know if the descrepency between estimated percentage is something to address from a UX perspective, something to consider messaging the user on initial snapshot load:

  • I see the block status widget updated after snapshot loading is completed.

Initial snapshot load: Progress is calculated from the snapshot point, so it looks high.

  • then I quit the app, and relaunched it....the block status widget not on the same page anymore...

After restart: Progress is recalculated from the beginning of the chain, so it drops.

This is expected behavior and will resolve itself as the node downloads and verifies the missing historical blocks.

  • go to Connection Settings, the Load snapshot option has disappeared (iirc this is expected behaviour) so I assume it still worked.

yes for this PR this is the expected behavior, in #446 instead of just disappearing it will be replaced by generate snapshot option

  • Did I test it right? Did I miss something I should've tested...lemme know.

Yes! Thank you for testing and highlighting potential UX issues. Please let me know if there are any design adjustments that should be tracked (i.e. notify the user that the high progress number is only because they are in snapshot mode)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants